I’m unsure whether I should upvote this post or not. Much of it seems to raise valid points.
If the articles provides more good than bad (as in 10 well-articulated objections versus 1 short trolling paragraph), I guess it still deserves an upvote.
I hate karma games, and usually automatically downvote any article or comment that speaks about its own karma. But this article has enough useful content that I made an exception of this rule. (It also helped that the article is rather long, so I skipped the offending paragraph.)
I disagree with the idea that critisicm is downvoted here, unless it happens to be badly written criticism, and I consider this accusation very unfair. However I upvoted this article, not to provide some kind of “balance” or “diversity”, but as my honest judgement of quality of the first 90% of its text.
EDIT: I also very much liked the idea that a self-improving AI would probably wirehead itself. It never occured to me, and it makes a lot of sense. (However, if I hope that future humans will be able to resist wireheading, it makes sense to worry that some AIs will manage to resist wireheading too.)
I also very much liked the idea that a self-improving AI would probably wirehead itself. It never occured to me, and it makes a lot of sense.
This idea is intuitively plausible, but doesn’t hold up when considering rational actors that value states of the world instead of states of their minds. Consider a paperclip maximizer, with the goal “make the number of paperclips in the universe as great as possible”. Would it rather a) make paperclips, or b) wirehead to convince itself that the universe is already full of paperclips? Before it wireheads, it knows that option a) will lead to more paperclips, so it does that. Similarly, I would rather actually help people than feel the warm glow that comes from helping people without any actual helping.
Well, yes, but behind the scenes you need a sensible symbolic representation of the world, with explicitly demarcated levels of abstraction. So, when the system is pathing between ‘the world now’ and ‘the world it wants to get to,’ the worlds in which it believes there are a lot of paperclips are in very different parts of state space than the worlds which contain the most paperclips, which is what it’s aiming for. Being unable to differentiate would be a bug in the seed AI, one which would not occur later if it did not originally exist.
If the articles provides more good than bad (as in 10 well-articulated objections versus 1 short trolling paragraph), I guess it still deserves an upvote.
I hate karma games, and usually automatically downvote any article or comment that speaks about its own karma. But this article has enough useful content that I made an exception of this rule. (It also helped that the article is rather long, so I skipped the offending paragraph.)
I disagree with the idea that critisicm is downvoted here, unless it happens to be badly written criticism, and I consider this accusation very unfair. However I upvoted this article, not to provide some kind of “balance” or “diversity”, but as my honest judgement of quality of the first 90% of its text.
EDIT: I also very much liked the idea that a self-improving AI would probably wirehead itself. It never occured to me, and it makes a lot of sense. (However, if I hope that future humans will be able to resist wireheading, it makes sense to worry that some AIs will manage to resist wireheading too.)
This idea is intuitively plausible, but doesn’t hold up when considering rational actors that value states of the world instead of states of their minds. Consider a paperclip maximizer, with the goal “make the number of paperclips in the universe as great as possible”. Would it rather a) make paperclips, or b) wirehead to convince itself that the universe is already full of paperclips? Before it wireheads, it knows that option a) will lead to more paperclips, so it does that. Similarly, I would rather actually help people than feel the warm glow that comes from helping people without any actual helping.
Easier said than done. Valuing state of the world is hard; you have to rely on senses.
Well, yes, but behind the scenes you need a sensible symbolic representation of the world, with explicitly demarcated levels of abstraction. So, when the system is pathing between ‘the world now’ and ‘the world it wants to get to,’ the worlds in which it believes there are a lot of paperclips are in very different parts of state space than the worlds which contain the most paperclips, which is what it’s aiming for. Being unable to differentiate would be a bug in the seed AI, one which would not occur later if it did not originally exist.